IMAGE PROCESSING DEVICE AND CAMERA 



BACKGROUND OF THE INVENTION 
The present invention relates to an image processing device that processes a signal 
5 output from an image sensor and outputs the processed signal. 

In digital cameras that use a charge-coupled device (CCD), a complementary metal 
oxide semiconductor (CMOS) image pickup device or the like as the image sensor, such as 
digital still cameras, digital camera-equipped mobile phones and digital video cameras, an 
image processing device performs image processing for an image signal read from the 
10 image sensor and outputs the results to a display device. In this relation, reading of an 
image from the image sensor and outputting of the image signal to the display device are 
made every line of a frame. Accordingly, in many cases, each function block of the image 
processing device performs its processing using a line memory that can store data by line. 

With the increase of the number of pixels of image sensors, the number of pixels of 
15 one line has increased and this has raised the need for increasing the capacity of line 
memories. To respond to this need, there is disclosed an image processing device in which 
an external memory capable of storing data of one entire frame is used and image 
processing is performed by block (see Japanese Laid-Open Patent Publication No. 2000- 
354193, for example). 

20 FIG. 16 is a block diagram of an example of a conventional camera, showing a 

configuration disclosed in the publication described above. In FIG. 16, an image signal 
read from an image sensor by line is stored in an external memory 914. An image 
processing device (signal processing circuit) 916 performs image processing for the image 
signal stored in the external memory 914 by reading the image signal by block, and stores 

25 the processed signal again in the external memory 914. A display device reads the 
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processed signal from the external memory 914 for display. 

Recent digital cameras are requested to achieve various extended functions of 
scaling down an image, rotating an image and the like, in addition to the fiinction of taking 
pictures. They are also requested to provide a plurality of image-quality modes in 
5 consideration of the capacities of recording media. In other words, the image processing 
device is required to perform a plurality of processing items, and the capacity of the 
memory required and the way of use of the memory vary with the processing item. 

In the camera of FIG. 16, image processing is performed by block. Accordingly, 
four times of access to the external memory 914 are necessary whichever type of 
10 processing is to be performed. Specifically, access to the external memory is necessary 
when an image signal read fi-om the image sensor by line is written, when the image 
processing device reads the stored image signal by block, when the image-processed data 
is written by block, and when the stored image-processed signal is read for display. 

In general, access to an external memory consumes much power and requires long 
15 processing time, compared with access to an internal memory. Accordingly, a system as 
that described above that makes access to an external memory invariably during image 
processing has a problem that at least some amounts of power and processing time are 
necessary regardless of the type of the processing. 

In the processing by block, data of a pixel belonging to a plurality of blocks in 
20 duplication is read a plurality of times. Therefore, the number of times of access to the 
external memory increased compared with the case of reading data once for each pixel. 
This also causes a problem in power consumption and processing time. 

Image processing devices can be mounted in various systems such as digital still 
cameras, digital video cameras and digital camera-equipped mobile phones. The functions 
25 required for the image processing devices however differ among the systems. For 
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example, in digital 3till cameras, in which an image having a large number of pixels must 
be processed, extended functions such as on-screen display (OSD) processing and rotation 
are naturally required. In digital camera-equipped mobile phones, in which the number of 
pixels of an image processed is small, extended functions such as scaledown and rotation 
5 are not required in many cases. In this case, therefore, no external memory is necessary. 
However, in the prior art disclosed in the above publication, the image processing device 
inevitably requires the external memory regardless of the system for which the image 
processing device is used. 

Different image processing devices may be mounted in different systems. 
10 Recently, however, reduction in the development cost of image processing devices has 
been strongly requested. In this situation, it is desired that one type of image processing 
device can be used for a plurality of systems. 



SUMMARY OF THE INVENTION 

15 An object of the present invention is providing an image processing device capable 

of achieving both low power consumption and multi-function capability. Another object 
of the present invention is providing a highly versatile image processing device usable for 
various systems. 

Specifically, the present invention is directed to an image processing device for 
20 performing image processing for an image signal output from an image sensor and 
outputting the results, including: an internal memory having line memories for storing an 
image by row or column as a unit; an image processing section for performing the image 
processing using the internal memory; and a CPU for controlling the image processing 
section, wherein the image processing section includes a plurality of processing circuits 
25 each performing predetermined processing as the image processing, and at least one of the 
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plurality of processing cirQuits is configured to allow use of an external memory provided 
outside the image processing device as required. 

According to the invention described above, the image processing device can select 
either achieving reduction in power consumption and enhancement in processing speed by 
5 using the internal memory or realizing an extended function requiring a large capacity of 
memory by using the external memory. Therefore, when no external memory is provided, 
the image processing device is adapted to perform basic image processing while achieving 
reduced power consumption. When an external memory is provided, the image processing 
device is adapted to achieve multi-function capability. Such an image processing device 
10 can be mounted in various types of systems, and thus the development cost of systems can 
be reduced. 

Since the internal memory stores data by line, it is not necessary to store data in an 
external memory temporarily before being transferred to a display device. Therefore, in 
the entire system, reduction in power consumption and enhancement in processing speed 
15 can be achieved. 

In the image processing device described above, the image processing section 
preferably performs given processing as the image processing without use of the external 
memory when the given processing can be performed without use of the external memory. 

According to the invention described above, access to the external memory is made 
20 only when necessary. This enables reduction in power consumption and enhancement in 
the processing speed. 

In the image processing device described above, the CPU preferably outputs a 
control signal for reducing power consumed by the external memory when the image 
processing section does not use the external memory. 
25 According to the invention described above, power consumed by the external 
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memory can be reduced when no access to the external memory is made. This further 
reduces power consumption. 

In the image processing device described above, the image processing section 
preferably includes as the plurality of processing circuits: a luminance/color-difiference 
5 signal processing circuit for converting the image signal obtained from the image sensor to 
a luminance signal and a color-difference signal and outputting the converted signals; and 
a compression circuit for performing compression coding of an image corresponding to the 
output of the luminance/color-difference signal processing circuit and outputting the results 
as an output of the image processing section. 
10 In the image processing device described above, the image processing section 

preferably includes an on-screen display processing circuit for superimposing an image 
read from the external memory on the image obtained from the image sensor and 
outputting the results. 

In the image processing device described above, preferably, one of the plurality of 
15 processing circuits stores a processed image into the external memory sequentially by row 
or column as a unit, and another one of the plurality of processing circuits reads the image 
stored in the external memory by column or row as a unit whichever different from the unit 
used during the storing of the image. 

According to the invention described above, rotation of an image can be realized. 
20 In the image processing device described above, at least two of the plurality of 

processing circuits preferably perform processing using the same internal memory. 

According to the invention described above, the capacity of the internal memory 
can be reduced compared with the case of providing exclusive memories for the plurality 
of processing items. 

25 In the image processing device described above, preferably, the image processing 



section stores the image output from the image sensor into the external memory, reads the 
stored image from the external memory by row or column as a unit whichever has a 
smaller number of pixels, performs the image processing for the read data using the 
internal memory, stores the results into an area of the external memory in which the 
5 corresponding pixel data had been stored before being read, and reads the resultant image 
from the external memory to be output. 

According to the invention described above, image processing can be performed for 
an image of a landscape shape having a horizontal number of pixels exceeding the number 
of pixels of one row allowed to be stored in the internal memory. 
10 In the image processing device described above, preferably, the image processing 

section divides the image into a plurality of areas if the number of pixels of each row or 
column of the image whichever is smaller exceeds the number of pixels allowed to be 
stored in the internal memory. 

The camera of the present invention includes: the image processing device 
15 described above; an image sensor for outputting an image signal to the image processing 
device; and a recording device for writing an output of the image processing device into a 
recording medium. 

BRIEF DESCRIPTION OF THE DRAWINGS 

20 FIG. 1 is a block diagram of an example of a camera having an image processing 

device of an embodiment of the present invention. 

FIG. 2 is a block diagram of an example of an image processing section in FIG. 1. 
FIG. 3 is a flowchart showing a part of an example of a flow of processing 
performed by the image processing device in FIG. I. 
25 FIG. 4 is a flowchart showing another part of the example of the flow of processing 
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performed by the image processing device in FIG, 1. 

FIG. 5 is a flowchart showing the remaining part of the example of the flow of 
processing performed by the image processing device in FIG. 1. 

FIG. 6 is a view illustrating a first example of a flow of data in the image 
5 processing device in FIG. 1. 

FIG. 7 is a view illustrating a second example of a flow of data in the image 
processing device in FIG. 1. 

FIG. 8 is a view illustrating a third example of a flow of data in the image 
processing device in FIG. 1. 
10 FIG. 9 is a view illustrating a fourth example of a flow of data in the image 

processing device in FIG. 1. 

FIG. lOA is a view illustrating an example of the order of pixels written in rotation 

processing. 

FIG. 1 OB is a view illustrating an example of the order of pixels read in rotation 
15 processing. 

FIG. 11 is a flowchart showing transfer of pixel data in image processing followed 
when no external memory is used. 

FIG. 12 is a view illustrating an example of a flow of data in processing of an 
image having more than m pixels horizontally. 
20 FIG. 13A is a view illustrating an example of the order of pixels transferred, in the 

case that the horizontal number of pixels exceeds m. 

FIG. 13B is a view illustrating another example of the order of pixels transferred, in 
the case that the horizontal number of pixels exceeds m. 

FIG. 14 is a flowchart showing an example of a flow of processing performed by 
2 5 the image processing device in FIG. 1 in the case of processing an image having more than 
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tn pixels horizontally, 

FIG. 15 is a view illustrating an example of the order of pixels transferred, in the 
case that both the horizontal and vertical numbers of pixels exceed m. 

FIG. 16 is a block diagram of an example of a conventional camera. 

5 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Hereinafter, a preferred embodiment of the present invention will be described with 
reference to the accompanying drawings. 

FIG 1 is a block diagram of an example of a camera having an image processing 
10 device of an embodiment of the present invention. The camera of FIG 1 is a digital still 
camera, a digital camera-equipped mobile phone or a digital video camera, for example. 
The camera of FIG. 1 includes an image processing device 100, an image sensor 12, an AD 
converter (ADC) 13, a recording device 14, a display device 15, an external memory 16 
and a power supply circuit 17. The image processing device 100 includes an image 
15 processing section 20, a CPU 50, an internal memory 60 and an output section 70. Note 
that lines of pixels in the horizontal direction and those in the vertical direction are herein 
referred to as rows and columns, respectively. 

The image sensor 12, which is a CCD or a CMOS image pickup device, for 
example, outputs an image signal to the AD converter 13. The AD converter 13 converts 
20 the input signal to digital data and outputs the resultant data to the image processing 
section 20. 

The image processing section 20 performs image processing for the output of the 
image sensor 12 under instructions from the CPU 50 and outputs the results to the output 
section 70. In performing the image processing, the image processing section 20 uses at 
2 5 least one of the internal memory 60 and the external memory 16 depending on the type of 
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the processing. 

The internal memory 60 has a plurality of line memories for storing an image by 
row or column. Each line memory has a capacity permitting storage of data of m pixels (m 
is a natural number) (this capacity is referred to as IH). Since pixels of one row of an 
5 image are not stored over a plurality of line memories, m is the maximum number of pixels 
that can be stored in the internal memory 60 as one row of pixels. In general, therefore, m 
is the maximum number of pixels of one row of an image that can be processed by the 
image processing device 100. Assume herein that m = 1280 and the internal memory 60 
has 18 line memories, for example. 

10 The external memory 16 is used as a frame memory and has a capacity permitting 

storage of at least one frame of the largest image that can be processed by the image 
processing device 100. 

The power supply circuit 17 supplies power to the external memory 16 under 
instructions from the CPU 50. The output section 70 has a buffer and operates as an 

15 interface that converts the output of the image processing section 20 to signals in forms 
suitable for write into a recording medium in the recording device 14 and for display with 
the display device 15 and outputs the resultant signals. The recording device 14 writes the 
output of the output section 70 into the recording medium such as a memory card. The 
display device 15, which is a liquid crystal display, for example, displays an image output 

20 from the image sensor 12 for monitoring. 

FIG. 2 is a block diagram of an example of the image processing section 20 in FIG 
1. The image processing section 20 includes a preprocessing circuit 22, a luminance/color- 
difference signal processing circuit (YC signal processing circuit) 24, a zoom-out circuit 
26, a post-filter 28, an OSD processing circuit 32, a JPEG (joint photographic image 

2 5 coding experts group) processing circuit 34 and a vertical enlargement circuit 36. The 
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image processing section 20 further includes an internal memory control circuit 42 and an 
external memory control circuit 44. All of these circuits of the image processing section 20 
operate under instructions from the CPU 50. 

Assume that the preprocessing circuit 22, the YC signal processing circuit 24, the 
5 zoom-out circuit 26, the post-filter 28, the OSD processing circuit 32, the JPEG processing 
circuit 34 and the vertical enlargement circuit 36 are allowed to output received data as it is 
with no processing performed for the data. These circuits access the internal memory 60 
via the internal memory control circuit 42 and the external memory 16 via the external 
memory control circuit 44. Therefore, these circuits are allowed to pass data among 

10 themselves via the external memory 16. 

The preprocessing circuit 22, the YC signal processing circuit 24, the zoom-out 
circuit 26, the post-filter 28, the OSD processing circuit 32, the JPEG processing circuit 34 
and the vertical enlargement circuit 36 use the same internal memory in performing 
processing. To state differently, these processing circuits are configured to share the 

15 internal memory 60. The CPU 50 allocates areas of the internal memory 60 to these 
processing circuits according to the need of processing by the respective processing 
circuits. 

The preprocessing circuit 22 performs at least one of extraction of the black level, 
detection of the white balance and gamma correction as preprocessing for an image signal 
20 output from the image sensor 12, and outputs the results to the YC signal processing circuit 
24. 

The YC signal processing circuit 24 receives the output of the preprocessing circuit 
22, performs YC signal processing for the received signal, and outputs the results to the 
zoom-out circuit 26. The YC signal processing includes correction of the black level, 
25 correction of the white balance and conversion to a luminance signal and a color-difference 
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signal. 

The zoom-out circuit 26 performs zooming using linear interpolation to scale down 
an image represented by the luminance signal and the color-difference signal, and outputs 
the results to the post-filter 28. 
5 The post-filter 28, which has a variable-coefficient low-pass filter, performs post- 

filtering that includes allowing passing of a low-frequency component of the image 
received from the zoom-out circuit 26 and performing aperture correction, and outputs the 
resuhs to the OSD processing circuit 32. 

The OSD processing circuit 32 reads image data for OSD fi^om the external 
10 memory 16, performs OSD processing of superimposing the read data on the image 
received from the post-filter 28, and outputs the results to the JPEG processing circuit 34. 

The JPEG processing circuit 34 as a compression circuit performs JPEG-based 
compression coding (JPEG compression) for the image received from the OSD processing 
circuit 32, and outputs the results to the output section 70 via the vertical enlargement 
15 circuit 36. The JPEG compressed results are then sent to the recording device 14 to be 
written into a recording medium such as a memory card. 

The vertical enlargement circuit 36 receives the image output from the OSD 
processing circuit 32 via the JPEG processing circuit 34, vertically enlarges the received 
image so that the number of pixels in the vertical direction matches the display device 15, 
20 and outputs the results to the output section 70. The vertically enlarged results are then 
sent to the display device 15 to be displayed. 

FIGS. 3, 4 and 5 are flowcharts totally showing an example of a flow of processing 
performed by the image processing device 100 in FIG. 1, 

FIG 6 is a view illustrating a first example of a flow of data in the image 
2 5 processing device 100 in FIG 1. Note that the internal memory control circuit 42 and the 
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external memory circuit 44 are omitted in FIG. 6 and similar figures to follow illustrating a 
flow of data. In the example of FIG. 6, assume that the size of an image output from the 
image sensor 12 is 1280 (= m) (horizontal) x 960 pixels (vertical), and that preprocessing, 
YC signal processing, zooming (2/3 X in this example), post-filtering and JPEG 
5 compression are performed as the image processing. In this case, the data amount of one 
horizontal row of an image input into the image processing device 100 corresponds to IH. 
The operation of the image processing device 100 in this example will be described with 
reference to FIGS. 2 to 6. 

In step Sll in FIG 3, the CPU 50 sets the horizontal number of pixels of an image 

10 represented by a signal output from the image sensor 12 as the horizontal number of pixels 
of an image input into the image processing device 100. This value can be set fi-om outside 
the image processing device 100 based on the type of the image sensor 12 connected. 
More specifically, it is set whether or not the horizontal number of pixels of an image 
handled by the image processing device 100 is 1/2 or less of the number of pixels (= m) 

15 allowed to be stored in each line memory of the internal memory 60. 

If the horizontal number of pixels is m/2 or less, pixel data of two rows can be 
stored in one line memory, and this allows much processing to be performed using only the 
internal memory 60. If the horizontal number of pixels is more than m/2, pixel data of 
only one row can be stored in one line memory, and this Umits the processing allowed to be 

20 performed using only the internal memory 60, Therefore, depending on whether or not the 
horizontal number of pixels is m/2 or less, it is necessary to change the capacities of areas 
of the internal memory 60 allocated to the respective processing circuits. In the illustrated 
example, in which m = 1280, it is set that the horizontal number of pixels is more than m/2. 
In step S12, the CPU 50 sets processing details. Specifically, set are whether or not 

25 zooming, post-filtering, JPEG compression, vertical enlargement, OSD processing. 
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rotation and the like are performed, together with the scaling factor of the zooming, if the 
zooming is performed, and the like. In the example of FIG. 6, it is set that zooming, post- 
fihering and JPEG compression are performed and that the scaling factor of the zooming is 
2/3. 

5 In step S13, the CPU 50 allocates areas of the internal memory 60 and the external 

memory 16 to the processing circuits of the image processing section 20, that is, the 
preprocessing circuit 22, the YC signal processing circuit 24, the zoom-out circuit 26, the 
post-filter 28, the OSD processing circuit 32, the JPEG processing circuit 34 and the 
vertical enlargement circuit 36, It is therefore made possible for each processing circuit to 

10 select whether to perform its processing using only the internal memory 60 or using the 
external memory 16 additionally, as required. 

The allocation of the internal memory 60 is made for only the processing items to 
be actually performed. An area of the external memory 16 is allocated to the OSD 
processing circuit 32 because OSD processing invariably requires use of the external 

15 memory 16. Rotation processing, which is to be discussed later, also requires use of the 
external memory 16, and therefore an area of the external memory 16 is allocated for this 
processing. In the case of performing zooming with a scaling factor of 1/2 or less, for 
example, pixel data of a plurality of rows can be stored in one line memory in the 
subsequent processing. This point is also considered in the allocation. 

20 In the example of FIG. 6, in which the scaling factor of the zooming is 2/3, data of 

only one row of a scaled-down image can be stored in one line memory. In this case, if 
JPEG compression is performed using the internal memory 60, a line memory of 8H will 
be necessary. Considering the capacity (18H) of the internal memory 60, therefore, it is 
decided that line memories corresponding to 2H, 4H, 4H, and 4H of the internal memory 

25 60 are allocated to the preprocessing circuit 22, the YC signal processing circuit 24, the 
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zoom-out circuit 26. and the post-filter 28, respectively, and an area of the external memory 
16 is allocated to the JPEG processing circuit 34. 

In step S16, the CPU 50 determines whether or not the external memory 16 is used, 
considering whether or not a processing item requiring use of the external memory 16 is 
5 performed, or whether or not an external memory is in connection, for example. If the 
external memory 16 is used, the process proceeds to step S22, Otherwise it proceeds to 
step S18. In the example of FIG. 6, in which the JPEG processing circuit 34 uses the 
external memory 16, the process proceeds to step S22. 

In step S22, the CPU 50 determines whether or not preprocessing is performed 
10 using only the internal memory 60. If only the internal memory 60 is used, the process 
proceeds to step S23. Otherwise it proceeds to step S24. In the example of FIG. 6, the 
process proceeds to step S23. 

In step S23, the preprocessing circuit 22 performs preprocessing while conducting 
read/write with the area of the internal memory 60 allocated to this circuit, handling each 
15 horizontal row of an image represented by an image signal output from the image sensor 
12 as one unit, and outputs the results to the YC signal processing circuit 24. The process 
then proceeds to step S27. 

In step S24, as in step S23, the preprocessing circuit 22 performs preprocessing 
while conducting read/write with the area of the external memory 16 or the internal 
20 memory 60 allocated to this circuit, handling each horizontal row of an image as one unit, 
and outputs the results to the YC signal processing circuit 24. The process then proceeds 
to step S27. 

In step S27, the CPU 50 determines whether or not YC signal preprocessing is 
performed using only the internal memory 60. If only the internal memory 60 is used, the 
25 process proceeds to step S28. Otherwise it proceeds to step S29. In the example of FIG 6, 
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the process proceeds to step S28. 

In step S28, the YC signal processing circuit 24 performs YC signal processing 
while conducting read/write with the area of the internal memory 60 allocated to this 
circuit, and outputs the results to the zoom-out circuit 26. The process then proceeds to 
5 step S31. 

In step S29, the YC signal processing circuit 24 performs YC signal processing 

while conducting read/write with the area of the external memory 16 or the internal 

memory 60 allocated to this circuit, and outputs the results to the zoom-out circuit 26. The 

process then proceeds to step S31. 
10 In step S31, the CPU 50 determines whether or not zooming is performed. If 

zooming is performed, the process proceeds to step S32. Otherwise it proceeds to step S36. 

In the example of FIG 6, in which zooming is performed, the process proceeds to step S32. 
In step S32, the CPU 50 determines whether or not zooming is performed using 

only the internal memory 60, If only the internal memory 60 is used, the process proceeds 
15 to step S33. Otherwise it proceeds to step S34. In the example of FIG 6, the process 

proceeds to step S33. 

In step S33, the zoom-out circuit 26 performs zooming of reducing the number of 
pixels of an image while conducting read/write with the area of the internal memory 60 
allocated to this circuit, and outputs the results to the post-filter 28. The process then 
2 0 proceeds to step S36. In the example of FIG 6, the zoom-out circuit 26 scales down the 
image so that the horizontal number of pixels is reduced to 2/3. 

In step S34, the zoom-out circuit 26 performs zooming while conducting read/write 
with the area of the external memory 16 or the internal memory 60 allocated to this circuit, 
and outputs the results to the post-filter 28. The process then proceeds to step S36. 
25 In step S36 in FIG 4, the CPU 50 determines whether or not post-filtering is 
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performed. If post-filtering is performed, the process proceeds to step S37. Otherwise it 
proceeds to step S41. In the example of FIG 6, in which post-filtering is performed, the 
process proceeds to step S37. 

In step S37, the CPU 50 determines whether or not post-filtering is performed using 
5 only the internal memory 60. If only the internal memory 60 is used, the process proceeds 
to step S38. Otherwise it proceeds to step S39. In the example of FIG 6, the process 
proceeds to step S38. 

In step S38, the post-filter 28 performs post-filtering while conducting read/write 
with the area of the internal memory 60 allocated to the post-filter, and outputs the results 
10 to the OSD processing circuit 32. The process then proceeds to step S41. 

In step S39, the post-filter 28 performs post-filtering while conducting read/write 
with the area of the external memory 16 or the internal memory 60 allocated to the post- 
fiher, and outputs the resuhs to the OSD processing circuit 32. The process then proceeds 
to step S41. 

15 In step S41, the CPU 50 determines whether or not OSD processing is performed. 

If OSD processing is performed, the process proceeds to step S44. Otherwise it proceeds 
to step S51. In the example of FIG 6, in which no OSD processing is performed, the 
process proceeds to step S51. 

In step 844, the OSD processing circuit 32 reads image data for OSD fi-om the 

20 external memory 16, performs OSD processing of superimposing the read data on the 
image received fi-om the post-filter 28 using the area of the external memory 16 or the 
internal memory 60 allocated to this circuit, and outputs the results to the JPEG processing 
circuit 34. The process then proceeds to step 851. 

In step 851, the CPU 50 determines whether or not JPEG compression is performed. 

25 If JPEG compression is performed, the process proceeds to step S52. Otherwise it 
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proceeds to step S56. In the example of FIG 6, in which JPEG compression is performed, 
the process proceeds to step S52. 

In step S52, the CPU 50 determines whether or not JPEG compression is performed 
using only the internal memory 60. If only the internal memory 60 is used, the process 
5 proceeds to step S53, Otherwise it proceeds to step S54. In the example of FIG 6, in 
which the external memory 16 is allocated to the JPEG processing circuit 34, the process 
proceeds to step SS4. 

In step S53, the JPEG processing circuit 34 performs JPEG compression while 
conducting read/write with the area of the internal memory 60 allocated to this circuit, and 
10 outputs the results to the output section 70 via the vertical enlargement circuit 36. The 
process is then terminated. 

In step S54, the JPEG processing circuit 34 performs JPEG compression while 
conducting read/write with the area of the external memory 16 or the internal memory 60 
allocated to this circuit, and outputs the results to the output section 70 via the vertical 
15 enlargement circuit 36. The process is then terminated. 

In step S56, the CPU 50 determines whether or not vertical enlargement is 
performed. If vertical enlargement is performed, the process proceeds to step S57. 
Otherwise the process is terminated. 

In step S57, the CPU 50 determines whether or not vertical enlargement is 
20 performed using only the internal memory 60. If only the internal memory 60 is used, the 
process proceeds to step S58. Otherwise it proceeds to step S59. 

In step S58, the vertical enlargement circuit 36 performs vertical enlargement while 
conducting read/write with the area of the internal memory 60 allocated to this circuit, and 
outputs the results to the output section 70. The process is then terminated, 
2 5 In step S59, the vertical enlargement circuit 36 performs vertical enlargement while 
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conducting read/write with the area of the external memory 16 or the internal memory 60 
allocated, to this circuit, and outputs the results to the output section 70. The process is 
then terminated. 

As described above, in the example of FIG 6, the external memory 16 is accessed 
5 only during the JPEG compression by the JPEG processing circuit 34. The other items of 
the image processing can be performed without use of the external memory 16, and 
therefore the external memory 16 is not used in these cases. This greatly reduces the 
access to the external memory 16. 

If the external memory 16 is not used at all, in step S18 in FIG 5, the CPU 50 
10 performs processing for shifting to a low power mode. For example, since the external 
memory 16 is no more used in the subsequent process steps, the CPU 50 halts supply of 
power to the external memory 16 by controlling the power supply circuit 17, to prevent the 
external memory 16 from consuming power. 

The subsequent series of processing in FIG 5 are similar to those described above 
15 with reference to FIGS. 3 and 4, except that the processing items are invariably performed 
using the internal memory with no determination on whether or not the internal memory is 
used, and that the step of asking whether or not OSD processing is performed and the step 
of performing OSD processing are not included. Specifically, steps S63, S68, 871, S73, 
S76, S78, S81, 883, 886 and 888 are respectively the same as steps 823, 828, 831, 833, 
20 836, 838, 851, 853, 856 and 858, and thus the description thereof is omitted here. In this 
way, processing of one fi"ame is terminated. 

As described above, the image processing device 100 performs either the JPEG 
compression or the vertical enlargement. Also, the image processing device 100 halts 
supply of power to the external memory 16 when the external memory 16 is not used, to 
25 prevent the external memory 16 from consuming power. 
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FIG 7 is a view illustrating a second example of a flow of data in the image 
processing device 100 in FIG. 1. In the example of FIG 7, as in the example of FIG 6, 
assume that the size of an image output from the image sensor 12 is 1280 (= m) 
(horizontal) x 960 pixels (vertical). In this example, assume that preprocessing, YC signal 
5 processing, zooming (1/4X in this example), post-filtering and JPEG compression are 
performed as the image processing. The operation of the image processing device 100 in 
this example will be described with reference to FIGS. 2 to 5 and 7. 

The processing in steps Sll and S12 is substantially the same as that in the example 
of FIG 6, except that the scaling factor of the zooming is set at 1/4. 

10 In the example of FIG 7, the horizontal number of pixels of an image input into the 

image processing device 100 is m and the scaling factor of the zooming is 1/4. 
Accordingly, one line memory can store data of four rows of a scaled-down image. In step 
S13, therefore, the CPU 50 allocates line memories corresponding to 2H, 4H, 4H, IH and 
2H of the internal memory 60 to the preprocessing circuit 22, the YC signal processing 

15 circuit 24, the zoom-out circuit 26, the post-filter 28 and the JPEG processing circuit 34, 
respectively. The capacity of the internal memory 60 is sufficient for this allocation. 
Therefore, no allocation of the external memory 16 is made. 

In step S16, the CPU 50 determines that the external memory 16 is not used, and 
the process proceeds to step S18. In step S18, supply of power to the external memory 16 

20 is halted. The series of processing in and after step S63 are the same as those in the 
example of FIG 6, except that each processing circuit invariably performs its processing 
while conducting read/write with the internal memory 60 with no determination by the 
CPU 50 on whether or not the internal memory 60 is used for the processing. 

As described above, the image processing device 100 can perform basic image 

25 processing even when no external memory is provided as shown in FIG 7. With no access 
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to the external memory 16 during image processing, the power consumption can be 
reduced and the processing speed can be enhanced. 

FIG 8 is a view illustrating a third example of a flow of data in the image 
processing device 100 in FIG 1. In the example of FIG 8, assume that the size of an 
5 image output from the image sensor 12 is 640 (= m/2) (horizontal) x 480 pixels (vertical), 
and that preprocessing, YC signal processing, zooming (1/2X in this example), post- 
filtering, OSD processing and JPEG compression are performed as the image processing. 
In this case, the data amount of one horizontal row of an image input into the image 
processing device 100 corresponds to 1/2H. The operation of the image processing device 

10 100 in this example will be described with reference to FIGS. 2 to 4 and 8. 

In step Sll, the CPU 50 sets that the horizontal number of pixels of an image input 
into the image processing device 100 is m/2 or less. In step S12, the CPU 50 sets that 
zooming, post-filtering, OSD processing and JPEG compression are performed and that the 
scaling factor of the zooming is 1/2. 

15 In the example of FIG 8, the horizontal number of pixels of an image input into the 

image processing device 100 is m/2 and the scaling factor of the zooming is 1/2. 
Accordingly, data of four rows of a scaled-down image can be stored in one line memory. 
In consideration of this, in step 813, the CPU 50 allocates line memories corresponding to 
IH, 2H, 2H, IH, IH and 2H of the internal memory 60 to the preprocessing circuit 22, the 

20 YC signal processing circuit 24, the zoom-out circuit 26, the post-filter 28, the OSD 
processing circuit 32 and the JPEG processing circuit 34, respectively. The capacity of the 
internal memory 60 is sufficient for this allocation. Therefore, no allocation of the external 
memory 16 is necessary. 

In step S16, the CPU 50 determines that the external memory 16 is used because 

25 OSD processing is to be performed, and the process proceeds to step S22. 
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The series of processing in and after step S22 in FIGS. 3 and 4 are the same as 
those in the example of FIG 6, except that in step S41, the CPU 50 determines that OSD 
processing is performed and proceeds to step S44, and that in step S44, the OSD 
processing circuit 32 performs OSD processing and outputs the results to the JPEG 
5 processing circuit 34. Detailed description of these steps is therefore omitted here. 

As described above, by use of the external memory 16, OSD processing can be 
performed as an extended function. Also, in the example of FIG 8, access to the external 
memory 16 is only once, which is made by the OSD processing circuit 32 for reading data. 
The internal memory 60 is used for the other cases of using a memory. This greatly 
10 reduces access to the external memory 16. 

FIG 9 is a view illustrating a fourth example of a flow of data in the image 
processing device 100 in FIG 1. In the example of FIG 8, assume that the size of an 
image output from the image sensor 12 is 1280 (= m) (horizontal) x 960 pixels (vertical), 
and that preprocessing, YC signal processing, zooming (1/2X in this example), post- 
15 filtering, vertical enlargement and rotation are performed as the image processing. The 
operation of the image processing device 100 in this example will be described with 
reference to FIGS. 2 to 4 and 9. 

In step Sll, the CPU 50 sets that the horizontal number of pixels of an image input 
into the image processing device 100 exceeds mil. In step 812, the CPU 50 sets that 
20 zooming, post-filtering, vertical enlargement and rotation are performed and that the 
scaling factor of the zooming is 1/2. 

In the example of FIG 9, the horizontal number of pixels of an image input into the 
image processing device 100 is m and the scaling factor of the zooming is 1/2. 
Accordingly, data of two rows of a scaled-down image can be stored in one line memory. 
2 5 In consideration of this, in step S13, the CPU 50 allocates line memories corresponding to 
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2H, 4H, 4H, 2H and 2H of the internal memory 60 to the preprocessing circuit 22, the YC 
signal processing circuit 24, the zoom-out circuit 26, the post-filter 28 and the vertical 
enlargement circuit 36, respectively. The capacity of the internal memory 60 is sufficient 
for this allocation. Therefore, no allocation of the external memory 16 is performed. 
5 In step S16, the CPU 50 determines that the external memory 16 is used because 

rotation must be performed, and the process proceeds to step S22. 

The series of processing by the preprocessing circuit 22, the YC signal processing 
circuit 24, the zoom-out circuit 26 and the post-filter 28 in the subsequent steps are 
substantially the same as those in the example of FIG 6, except that the scaling factor of 

10 the zooming is different. Note however that the YC signal processing circuit 24 and the 
zoom-out circuit 26 perform rotation using the external memory 16, although description 
on rotation is omitted in the flowchart in FIGS. 3 and 4. 

The rotation processing will be described. FIG. lOA is a view illustrating an 
example of the order of pixels written into the external memory 16 in rotation processing, 

15 and FIG lOB is a view illustrating an example of the order of pixels read from the external 
memory 16 in rotation processing. As shown in FIGS. lOA and lOB, an image output from 
the YC signal processing circuit 24 has m pixels horizontally and n pixels vertically. With 
coordinate values vl, v2, vn given to the horizontal rows and coordinate values hi, h2, 
. . . , hm given to the vertical columns, each pixel can be represented by the horizontal and 

20 vertical coordinates, like hlvl. 

The YC signal processing circuit 24 writes the YC-signal processed data of pixels 
into the external memory 16 by horizontal row of pixels in an image sequentially. 
Specifically, as shown in FIG. lOA, pixels of the first row are written in the order of hlvl, 
h2vl, hmvl, and then pixels of the second row are written in the order of hlv2, h2v2, 

25 hmv2. In this manner, pixels of the subsequent rows through the n-th row are 
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sequentially written. 

Thereafter, the zoom-out circuit 26 reads the data of pixejsjwritten by the YC signal 
processing circuit 24 from the external memory 16 by vertical column of pixels in the 
image as it was before being written. Specifically, as shown in FIG lOB, pixels of the first 
5 column are read in the order of hlvl, hi v2, . . hlvn, and then pixels of the second column 
are read in the order of h2vl, h2v2, h2vn. In this manner, pixels of the subsequent 
columns through the m-th column are sequentially read. 

The data of pixels of each column read in the above processing is handled as data 
of pixels of each horizontal row. In this way, an image rotated by 90*^ from the image as it 
10 was before being written in the external memory 16 is obtained. 

The zoom-out circuit 26 then performs zooming for the rotated image by reducing 
the horizontal number of pixels of the image to 1/2, and outputs the results to the post-filter 
28. After processing by the post-filter 28, the vertical enlargement circuit 36 performs 
vertical enlargement using the internal memory 60 and outputs the results to the output 
15 section 70, 

As described above, by use of the external memory 16, rotation of an image can be 
easily performed as an extended fiinction. 

Data of pixels may be written into the external memory 16 by vertical column of 
pixels in an image as shown in FIG. lOB, and thereafter, be read from the external memory 
20 16 by horizontal row of pixels in the image as it was before being written. 

Any of the preprocessing circuit 22, the YC signal processing circuit 24, the zoom- 
out circuit 26 and the post-filter 28 may write data into the external memory 16, and the 
circuit that is to perform processing next may read the data from the external memory 16, 
In this case, however, the read of the data from the external memory 16 must have been 
25 finished before the JPEG processing circuit 34 receives data. 
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Although already described with reference to FIG. 7, transfer of pixel data followed 
when the external memory 16 is not used is summarized as follows. FIG. 11 is a flowchart 
showing transfer of pixel data in image processing followed when the external memory 16 
is not used. 

5 In step S112, the CPU 50 stores pixel data of one row into the internal memory 60. 

In step S114, the image processing section 20 performs various types of image processing 
using the internal memory 60. In step S116, the image processing section 20 outputs the 
image-processed pixel data of one row to the output section 70. In step S118, the CPU 50 
determines whether or not processing of one frame has been completed. If completed, the 

10 process is terminated. Otherwise, the process returns to step S112. 

Since pixels of one row of an image are not stored over a plurality of line memories 
in the internal memory 60, m is the maximum number of pixels allowed to be stored in the 
internal memory as one row of pixels. Therefore, if the external memory 16 is not used, 
the image processing device 100 can process only an image having m or less pixels 

15 horizontally. 

By using an external memory, processing of an image having more than m pixels 
horizontally is made possible. Hereinafter, discussed will be the case that the image 
processing device 100 processes an image having more than m pixels horizontally using an 
external memory. 

20 FIG 12 is a view illustrating an example of a flow of data in processing of an image 

having more than m pixels horizontally. FIGS. 13A and 13B are views illustrating 
examples of the order of pixels transferred, in the case that the horizontal number of pixels 
exceeds m. The image shown in FIGS. 13A and 13B has m+a pixels (a is a natural 
number) horizontally and m pixels vertically. Coordinate values vl, v2, vm are given 

25 to the respective horizontal rows, and coordinate values hi, h2, . .., h(m+a) are given to the 
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respective vertical columns. 

Image processing for an image having more than m pixels horizontally is not 
allowed because the internal memory 60 fails to store pixel data of one row of such an 
image in one line memory. However, as for the image in FIG. 13A, since this image has m 
5 pixels vertically, processing can be made by storing pixels of one column in one line 
memory. 

FIG. 14 is a flowchart showing an example of a flow of processing performed by 
the image processing device 100 in FIG 1 in the case of processing an image having more 
than m pixels horizontally. The processing in this case will be described with reference to 
10 FIGS. 12 to 14. 

In step SI 22 in FIG 14, the image processing section 20 stores pixel data output 
from the AD converter 13 into the external memory 16 by row of pixels (Wl in FIG. 12). 
Specifically, as shown in FIG. 13A, pixels of the first row are written in the order of hlvl, 
h2vl, h(m+a)vl and then pixels of the second row are written in the order of hlv2, 
15 h2v2, . . h(m+a)v2. The write continues in this way for the subsequent rows through the 
m-th row. 

In step S124, the CPU 50 determines whether or not the number of pixels in the 
vertical direction (vertical number of pixels) of an image is m or less. If the vertical 
number of pixel is m or less, the process proceeds to step S132. Otherwise it proceeds to 
20 step S152. In the case of the image in FIG 13A, the process proceeds to step S132. 

In step S132, the image processing section 20 reads pixel data of one column from 
the external memory 16 (Rl in FIG 12) and transfers the data to the internal memory 60 
for storage. That is, pixels are read in the order of hlvl, hlv2, . . ., hlvm as shown in FIG 
13B, for example. 

25 In step S134, the image processing section 20 performs image processing such as 
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preprocessing, YC signal processing, zooming and post-filtering using the internal memory 
60 to obtain the pixel data of one column. 

In step SI 36, the image processing section 20 stores the processed pixel data of one 
column into the external memory 16 (W2 in FIG. 12). In this storage, the processed pixel 
5 data of one column is written in the area in which the corresponding pixel data of one 
column before the processing had been stored. 

In step S138, the CPU 50 determines whether or not processing of one frame has 
been completed. If completed, the process proceeds to step S142, Otherwise it returns to 
step S132. 

10 In step SI 42, the image processing section 20 reads pixel data of the resultant 

image from the external memory 16 by row of pixels as shown in FIG. 13A, for example 
(R2 in FIG 12). If further processing such as JPEG compression is required, such 
processing is performed for the read image, and the results are output to the output section 
70. 

15 As described above, by use of an external memory, the image processing device 

100 adapted to process an image having a horizontal number of pixels of m or less can 
perform various types of image processing for even an image having a horizontal number 
of pixels exceeding m without the necessity of changing the circuit configuration as long as 
the vertical number of pixels of the image is m or less. 

20 Hereinafter, processing for an image having both horizontal and vertical numbers 

of pixels exceeding m will be described. FIG 15 is a view illustrating an example of the 
order of pixels transferred, in the case that both the horizontal and vertical numbers of 
pixels of an image exceed m. The image in FIG 15 has m+a+p pixels (P is a natural 
number) horizontally and m+a pixels vertically. Coordinate values vl, v2, v(m+a) are 

25 given to the respective horizontal rows, and coordinate values hi, h2, h(m+a+p) are 
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given to the respective vertical columns. 

The processing in the above case will be described with reference to FIG. 14. In 
step S122, the CPU 50 stores pixel data output from the AD converter 13 into the external 
memory 16 by row of pixels. Specifically, pixels of the first row are written in the order of 
5 hlvl, h2vl, h(m+a+P)vl, and the write continues in this way for the subsequent rows 
through the (m+a)th row. 

In the case of the image in FIG 15, it is determined that the vertical number of 
pixels is not m or less in step S124, and the process proceeds to step S152. 

In step S152, the CPU 50 divides the image in FIG 15 into area a including the first 
10 to m-th rows and area b including the other rows. 

In step SI 54, the CPU 50 selects which area is to be processed, area a or area b. 
Also, the image processing device 100 reads pixel data of one column from the external 
memory 16 and transfers the data to the internal memory 60 for storage. That is, as shown 
in FIG 15, if area a is selected, pixels are read in the order of hlvl, hlv2, hlvm, for 
15 example. If area b is selected, pixels are read in the order of hlv(m+l), hlv(m+2), 
hi v(m+a), for example. 

The processing details in steps S156 and S158 are the same as those in steps S134 
and S136, although in steps S156 and S158, the processing is performed only for the 
selected area. 

20 In step S162, the CPU 50 determines whether or not processing in the selected area 

has been completed. If completed, the process proceeds to step S164. Otherwise-it returns 
to step S154 for processing of the remaining area. 

In step S164, the CPU 50 determines whether or not processing of one frame has 
been completed. If completed, the process proceeds to step S142. Otherwise it returns to 

2 5 stepS152. 
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As described above with reference to FIGS. 14 and 15, by use of an external 
memory, the image processing device 100 adapted to process an image having a horizontal 
number of pixels of m or less can perform various types of image processing for even an 
image having both vertical and horizontal numbers of pixels exceeding m without the 
5 necessity of changing the circuit configuration. 

As described in the above embodiment, the image processing device 100 in FIG. 1 
can perform image processing in both the cases of using an external memory and using no 
external memory. 

In the case that the external memory 16 is not provided, the image processing 
10 device 100 can perform basic image processing such as YC signal processing and 
compression. In this case, with no access to the external memory 16, the power 
consumption can be reduced and the processing speed can be enhanced. 

In the case that the external memory 16 is provided, it is possible to perform OSD 
processing, rotation, processing of a large image having a vertical or horizontal number of 
15 pixels exceeding m and the like. 

In the above description, the AD converter 13 was provided outside the image 
processing device 100. Alternatively, the image processing device may include the AD 
converter 13, 

Thus, according to the present invention, it is possible to realize an image 
20 processing device capable of achieving both low power consumption and multi-function 
capability In addition, it is possible to realize a highly versatile image processing device 
that is operable regardless of the presence or absence of an external memory and thus 
usable for various systems. 

While the present invention has been described in a preferred embodiment, it will 
25 be apparent to those skilled in the art that the disclosed invention may be modified in 
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numerous ways and may assume many embodiments other than that specifically set out 
and described above. Accordingly, it is intended by the appended claims to cover all 
modifications of the invention which fall within the true spirit and scope of the invention. 
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